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ABSTRACT 

This topical review addresses performance goals and 
indicators for special education, including how to create valid and 
measurable indicators of key features of special education that can be used 
to gauge effectiveness as well as target improvement strategies. A section 
identifying highlights of the review notes that an indicator system consists 
of four elements: context, input, process, and outcome-results. Following a 
brief introduction, the review is organized into six sections. Section 1 
reviews current accountability strategies and the features of an educational 
accountability system. Section 2 contains an overview of educational 
indicators and their use in accountability systems. Section 3 presents 
examples of specific indicator systems that are currently required in general 
education. Section 4 discusses special education data currently being 
collected under new federal education accountability systems. Section 5 is a 
summary and analysis of key issues to be considered in developing * performance 
goals and a special education indicator system. Section 6 offers practical 
steps for creating a special education indicator system. Throughout the 
review, examples are offered from the four states (California, Maryland, New 
York, and Texas) participating in the investigation of accountability in 
special education. (Contains 53 references.) (DB) 
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Topical Review Highlights 




T he demand for accountability is among the most visi- 
ble and controversial of US. educational reforms. The 
Federal government, states, and local districts increasingly 
use educational data to gauge the productivity of school sys- 
tems and to bold schools, districts, teachers, and students 
responsible for their performances and for meeting goals. 

Indicator Systems as an Element of Accountability 

Indicator systems are a key element of accountability, 
although they are not the whole accountability system. 
Indicators provide information, which can range from stu- 
dent-level outcomes such as academic achievement, to 
school-level input such as teacher quality and class size. In 
an effective accountability system, the information that 
indicators offer is used to monitor the " health ” of the educa- 
tion system, reported to appropriate agencies and the public, 
and acted upon. Among the indicators that states use most 
frequently are assessment scores, dropout rates, student 
attendance, and expenditures. 

An indicator system’s framework consists of 
four elements: 

• Context - the situation in which learning occurs, includ- 
ing school demographics and student-teacher 

interaction; 

• Input - human and fiscal resources, including personnel 
and facilities; 

• Process - school actions and materials that mediate 
between inputs and outcomes, including school 
operations and curriculum quality ; and 

• Outcome - results, including student achievement and 
graduation rates. 



Some education models try to determine the optimal 
proportion of inputs required to yield maximum outcomes: 
the "education production function. ” However, it’s impor- 
tant to know that indicator systems are fluid; one indicator 
may be labeled an "input’’ in one circumstance and a 
"process" or "outcome" in another. 

Examples of Federal Accountability Systems Requiring 
Indicators 

Two pieces of federal legislation requiring the use of 
indicators for enhanced accountability are the Government 
Performance Results Act (GPRA) and Title I of the Improving 
America’s Schools Act. GRPA bolds federal agencies account- 
able for producing results, and links plans and outcomes 
with budgets. Within the Department of Education, the Office 
of Special Education Programs (OSEP) has developed GPRA 
goals and indicators. 

Title I requires states to create - for all students - chal- 
lenging content and standards aligned with assessment and 
accountability systems. A state must show that it has met 
specific requirements and bos demonstrated adequate yearty 
progress (AYP) or be subject to corrective actions. However, 
Tide I requirements are interpreted in different ways by 
states; for example, states translate “assessments” to fit their 
own needs or existing systems ( Goertz & Duffy, 2000). In 
addition, 20 states have not met the Title I requirement to 
include students with disabilities or IEP students in their 
assessment and accountability systems (U.S. Department of 
Education, 2001). 
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4 Creating Performance Goals and Indicators in Special Education 



Special Education Accountability and Indicators 

Within special education, several efforts have been 
designed to provide OSEP with information on how relevant 
state agencies are complying with the 1997 amendments to 
IDEA. These efforts, which are performance-based, include 
the Continuous Improvement Monitoring Process (CIMP) 
and Focused Monitoring. 

IDEA requires that states establish performance goals 
and indicators for students with disabilities and report them 
to the public biennially. A review of those reports has found 
significant variations across states. For example, what one 
state called an indicator, another identified as a goal; some 
states had separate goals for students with disabilities and 
students in general education (Abeam, 2001). 



Issues in Creating Special Education Indicators 

An indicator system used to establish and monitor 
progress toward performance goals can be a powerful and 
controversial tool. To create an indicator system, policy- 
makers must first identify the areas needing improvement, 
then create benchmarks, or target performance levels (which 
can be either absolute standards or rates of progress). They 
should collect data on the present dimensions of the prob- 
lems being addressed and set goals that represent meaning- 
ful change in the direction of the benchmarks. 

The next step is to select key indicators that are accu- 
rate, efficient, timely, and valid. Since data is generally 
“reported up” from one level (student, school, district, state, 
or federal) to the next, policy-makers should create a system 
with vertical alignment, meaning that the data collected has 
use and significance at higher levels. Once policy-makers 
have chosen indicators, they need to monitor progress con- 
tinuously That way, they can evaluate targeted performance 
and intervene immediately to improve progress toward goals. 

Creating an effective, multi-use indicator system for 
special education presents challenges. However, the benefits 
of efficient and aligned data collection are clear in a climate 
calling for enhanced, performance-based accountability. 
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1 . Introduction 



This topical review addresses performance goals and indi- 
cators for special education. It is part of a series developed by 
the Educational Policy Reform Research Institute (EPRRI), 
which is investigating issues related to accountability and special 
education. This review looks at an area of increasing concern to 
special educators: how to create valid and measurable indica- 
tors of key features of special education that can be used to 
gauge effectiveness as well as target improvement strategies. 

The 1997 amendments to the Individuals with Disabilities 
Education Act (IDEA) requires states to set performance goals 
for children with disabilities that, to the extent appropriate, are 
consistent with goals for all children, and to establish indicators 
that can assess progress toward those goals. At a minimum, the 
goals must address performance on assessments, drop out 
rates, and graduation rates. This new requirement, coupled with 
additional changes in IDEA and other key federal legislation, 
such as Title 1 of the Elementary and Secondary Education Act, 
signal the national shift to performance-based accountability 
and the use of data-driven decision making. 

This review is organized in six sections. Section I presents a 
brief review of current accountability strategies and the features 
of an educational accountability system; Section 0 contains an 
overview of educational indicators and their use in accountabili- 
ty systems. Section ID presents examples of specific indicator 
systems that are currently required in general education. Section 
IV discusses special education data currently being collected. 
Section V is a summary and analysis of key issues to be consid- 
ered in developing performance goals and a special education 
indicator system. Section VI offers practical steps for creating a 
special education indicator system. 

Throughout this review we use examples from four states: 
California, Maryland, New York, and Texas. These are the core 
study states participating with EPRRI in the investigation of 
accountability in special education. The examples are not con- 
sidered representative of all states but are illustrative of specific 
types of reports and data being collected. 
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2. An Overview of Accountability 



The use of educational data to indicate the productivity 
of school systems is a key feature of accountability strategies. 
Ever since the s eminal report, A Nation at Risk (from the 
National Commission on Excellence in Education, 1983), com- 
pared the performance of U.S. students with that of students in 
other educational systems around the world, many politicians, 
policy-makers, corporate leaders, and academicians have 
expressed concern that American public education is not pro- 
ductive. 

Productive school systems use resources effectively to gen- 
erate more positive outcomes; students leave schools well pre- 
pared with skills that allow them to compete in the labor market 
or pursue higher education. Students become internationally 
competitive on a global level at least in terms of performance on 
international assessments. The demand for more productive 
schools has brought about the current push for greater educa- 
tional accountability. 

The demand for school and school district accountability is 
among the most visible and controversial of U.S. educational 
reforms. Current educational accountability systems put most 
emphasis on student performance (Fuhrman, 1999; Heubert & 
Hauser, 1999; Thurlow, Nelson, Teelucksingh, &Ysseldyke, 

2000) and direct consequences to systems and students 
(Heubert & Hauser, 1999). 

Creating Data-driven Accountability Systems 

An accountability system is a set of mechanisms that pro- 
vides information to the public. It is based on the notion that 
public education is a rational system involving inputs and out- 
comes. The inputs (e.g., teachers, students, and facilities) have 
certain characteristics, qualifications, socio-economic status, 
and resources (e.g., funds) (Shavelson, McDonnell, & Oakes, 
1989). The outcomes include academic achievement, gradua- 
tion rates, and other student performance data. In order to 
adjust inputs, schools must have data on intended outcomes. In 
addition, educational accountability depends on assigning 
responsibility for clearly defined or measured outcomes. 

Accountability systems may serve a number of purposes. 
They can be used to: (1) diagnose a problem within a certain 
set of educational practices; (2) correct a set of morally wrong, 
economically inefficient, or ineffective practices; (3) generate a 

O 



set of incentives that motivate professionals and students to 
achieve their best; (4) provide information to the public at 
large, in keeping with a democratic tradition of openness; (5) 
provide a standard to compare schools’ productivity and per- 
formance; and (6) ensure consequences for the actions of 
schools and teachers. Darling-Hammond (1989) states that 
accountability also “encompasses how a school or school sys- 
tem hires, evaluates, and supports its staff; how it relates to stu- 
dents and parents; how it manages its daily affairs; how it makes 
decisions; how it ensures that the best available knowledge will 
be acquired and used; how it evaluates its own functioning, as 
well as students’ progress; how it tackles problems; and how it 
provides incentives for continual improvement” (p. 20). 

Levels of Accountability 

There are three levels of accountability in educational sys- 
tems, each having implications for the types of data that are 
required: (1) school and district accountability, (2) profession- 
al or teacher accountability, and (3) student accountability. 

A. School and District Accountability 

Traditionally, LEAs have been responsible for implementing 

state and federal directives. New accountability systems focus on 
individual schools, and require that school-level data, most 
notably on student achievement, be collected and reported. 

These data are frequently used to impose consequences on 
schools (Fuhrman, 1999; Linn, 2000; Olson, Bond, & Andrews, 

1999) . 

Student performance indicators are the most visible part of 
school-level accountability (Olson et al., 1999; Linn 2000). 
These indicators typically include student achievement on state 
and local assessments, attendance, graduation, and in some 
instances, post-secondary status (Linn, 2000; Thurlow et al., 

2000) . Despite the emphasis on student assessment results, nei- 
ther states nor local districts have completely eliminated other 
data from their accountability systems. For example, some states 
(Kansas, Michigan, Rhode Island) are using school and district 
inspections of teaching processes to both accredit schools and 
assist them in improving student performance (Fuhrman, 1999; 
Goertz & Friedman, 1996). 

B. Teacher Accountability 

The concept of holding individual teachers directly respon- 
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sible for student performance remains controversial. While most 
states require school districts to evaluate teachers on a regular 
basis, teacher compensation, tenure and licensure typically are 
not tied to student achievement A few states (e.g., Texas) are 
taking steps to include student performance in the evaluation of 
teachers. Some states use school-based accountability as a way 
of rewarding (Kentucky) or punishing (Texas) teachers for stu- 
dents’ performance (Thurlow et al., 2000). Fuhrman (1999) 
asserts that even when there is no direct teacher accountability, 
teachers are motivated by the monetary bonuses a school may 
receive and also by the desire to avoid sanctions. 

C. Student Accountability 

When students are held accountable, typically they are at 
the high school level, and accountability is based on the courses 
they take, rather than on their performance on assessments 
(Heubert & Hauser, 1999). All but four states prescribe a mini- 
mum number of courses high school students must take in spe- 
cific academic areas (Guy, Shin, Young, & Thurlow, 1999). In 
1998, most states required four years of English, three years of 
social studies, and two years each of mathematics and sdence 
to receive a high school diploma. States, however, generally do 
not specify exactly which courses (e.g., algebra, geometry, 
chemistry) students must take to meet their credit requirements 
(Guy et al., 1999)- Twelve states award an advanced or honors 
diploma to students who complete additional or advanced 
courses. 



Increasingly, many states assess wfaat their students have 
learned by the time they reach the 1 1th or 12th grade. The 
Council of Chief State School Officers’ national survey indicated 
that 18 states required students to pass a profidency test to 
receive a high school diploma (Olson, 1999). More and more, 
assessment tests focus on standards for 10th grade or higher 
(American Federation of Teachers, 1999) as states revise exist- 
ing high school assessments or develop new ones to measure 
more rigorous content (Fuhrman, 1999). See “Exit Documents 
for Students with Disabilities: An Overview of Historical and 
Legal Issues’’ www.eppri.org . 

Tests are also increasingly used as requirements for promo- 
tion from one grade to the next (Heubert & Hauser, 1999) • 
Proponents of increased test-based accountability believe that 
high educational performance is not possible without challeng- 
ing and explidt standards attached to rewards. Nonetheless, 
test-based accountability is controversial, as parents and the 
general publicquestion issues of test validity, costs, focus, and 
teaching to the tests. 
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Creating Per f or m ance Goals and Indicators in Special Education 



3. An Overview of Education Indicators 

and Indicator Systems 



The creation of appropriate indicator systems is a key part 
of an educational accountability system. Indicator systems pro- 
vide a variety of information to government officials, school 
administrators, school officials, principals, teachers, parents, 
and the public at large. Indicators may include student-level out- 
comes (e.g., academic achievement) or they may focus on 
school-level inputs (e.g., teacher quality, learning processes, 
class size, teacher-pupil ratio, textbooks) . Schools and school 
districts are currently judged against a multitude of indicators. 

In 1999, the indicators most frequently used by states were 
assessment scores (41 states); dropout rates (33 states); stu- 
dent attendance (29 states); expenditures and the use of 
resources (27 states); graduation rates (18 states); student 
behavior, discipline, truancy, expulsion and/or suspension (18 
states); and transition to post-secondary education or employ- 
ment after high school (16 states) (Education Commission of 
the State, 1999)- 

However, Darling-Hammond (1989) cautions that despite 
what many people think, indicators are not the accountability 
system. Indicator systems only provide information. In order for 
that information to translate into an accountability framework, 
several other factors have to be present, mainly a set of policies 
regarding how the indicators will be reported and used, and 
sufficient capacity to act upon the information. 



The Uses of Education Indicators 

An education indicator is an individual or composite 
statistic that relates to a basic construct in education and is 
useful in a policy context. The overriding purpose of indica- 
tors is to characterize the nature of a system through its 
components — how they are related and bow they change 
over time. This information can then be used to judge 
progress toward some goal or standard, against some past 
benchmark, or by comparison with data from some other 
institution or country ( Sbavelson & Oakes, 1991, p. 1). 

Goedegeburre and Overgaag (1991) identify four main uses 
of indicators. 

(1) Preparation for choice and action - When new pro- 
grams or activities are contemplated, indicators can be 
useful in understanding the context in which those pro- 
grams will be implemented. They can provide a known 
base for planning implementation, some indication of the 
resources already in place, and a framework for estimat- 
ing needed resources. 

(2) Evaluation of choice and action - Once new policies 
have been implemented, it is important to collect data to 
determine whether the system is responding as expected 
and to see where there are any unintended consequences. 
Conversely, indicators can signal the need for new poli- 
cies to change undesirable trends. 

(3) Routinization of choice and action - Some indicators 
can play a role in the regular distribution of resources 
(funding, the distribution of teachers to schools, etc.). 

(4) Assembling a comprehensive knowledge base - An indi- 
cator system can provide a whole picture rather than a 
piecemeal assessment, and can signal a problem in cer- 
tain areas of the educational system. The use of educa- 
tional indicators in this context is still evolving. 
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Goertz (1989) states that, “Education Indicator systems pro- 
vide information not only on the distinct components of the 
educational system, but also about how the relevant compo- 
nents work together to affect educational outcomes” (p. 4). 

A single indicator by itself cannot tell the story of what goes 
on within an educational system. Porter (1991) cautions that: 

Unfortunately, an indicator system designed today is likefy 
to place an inappropriate focus on today’s issues. Just as such 
a system would be Judged as inadequate against the concerns 
of 10 years ago, that same system may well miss the issues of 

greatest interest 10 years from now. A snapshot of school 

practice is not sufficient; assessments of change are needed 
(p.14). 

Oakes (1986) agrees stating, “Indicators must have the 
potential to identify new problems as well as to address old 
questions” (p. 3). 



The Framework of Indicator Systems 

A general framework for indicator systems involves four 
do mains of data or information: Context t Input): Process! 
Outcome 

A. Context 

Context is the situation in which learning occurs. There are 
four levels of context that need to be considered in any indicator 
framework: The learners-, their cognitive states and abilities; The 
classroom: student-teacher interactions; scheduling of classes; 
time spent on instruction; level of technology within the class- 
room; The school: the demographics of students at the school; 
how the school differs from others within the same geographic 
region; how it compares to schools across the nation; the goals 
of particular schools; The community : community characteris- 
tics such as socioeconomic status, the amount of social capital 
available within the community (extended community net- 
works); and parental education within the community. Some 
contextual factors may also be considered as inputs. 

B. Inputs and Outcomes 

The concept of educational inputs defined as human and 
fiscal resources and outcomes such as student achievement and 
performance reflects some of the earliest educational accounta- 
bility indicators. 
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Examples of Educational Inputs 

• Personnel: teachers, administrators, clerical and technical 
workers. 

• Qualifications of Personnel : certification, education, years 
of experience, subject area training. 

• Infrastructure: facilities, buildings, labs, media, technology, 
operating systems, transportation, books, supplies. 

• Community Resources: partnerships, parental education, 
parental involvement, social capital, cultural capital, 
human capital. 

• Financial Resources: money. 

• Student Characteristics: students’ prior academic achieve- 
ment, effort they put into learning, time spent doing home- 
work, peer influence. 



Examples of Educational Outcomes 

• Academic achievement. 

• Cognitive higher order learning. 

• Aspirations for higher educational attainment. 

• Participation in academic ventures. 

• Graduation rates. 

• Dropout rates. 

• Labor market earnings. 

• College attendance. 



Some educational researchers have attempted to simulate 
or model the educational process following models developed 
in the disciplines of economics and business. These models 
attempt to determine die optimal proportion of inputs required 
to yield maximum outcomes. This is referred to as the 
“Education Production Function.” 

Researchers have tried to isolate the effect of each one of 
the inputs on the desired outcome, but the body of literature 
remains conflicted as to which inputs make a significant impact 
on outcomes. 

The primary limitation of the education production function 
is that it only views simple input-outcome relationships. It does 
not simultaneously take into account the processes that occur 
within a school or the context within which the inputs are situat- 
ed, nor does it account for how the input gets transformed into 
the outcome. 
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Modeling the Education Production Function 
Researchers have attempted to model real life phenomena to 
understand how various educational variables affect each 
other. By isolating variables that have a substantial impact, die 
model can yield important information for policy discussions 
or resource allocation. 

The basic form of the Education Production Function Model 
as defined by Taylor (1998) is: 

Yt = f( a (f), i(t-r), F(t-r), p(t-r), s(t-r), c(t-r), M(t-r)) 

The educational outcome of interest Y(t) measured at time t. 
This outcome is a function of all of the following inputs: 

• A(t*) is the student’s prior achievement, 

• I(t-t*) is the difference in the student’s effort between 
time t and t*, 

• F(t-t*) is the influence of the student’s family between 
time t and t*, 

• P(t-t*) is the influence of peers between time t and t*, 

• S(t-t*) is the influence of school between time t and t*, 

• C(t-t*) is the influence of community between 
time t and t*, and 

• M(t-t*) is the influence of financial resources between 
time t and t*. 



As the demand for accountability has risen, policy makers 
have tried to understand why some schools produce good 
results despite a low level of resources or inputs, while others 
produce lower results despite a high level of resources. There 
has been an increased recognition of the need to incorporate 
process indicators into the education production function 
(Oakes, 1986; Porter, 1991). 

C. Process Indicators 

School processes are those actions and materials which 
mediate the interaction between inputs and outcomes. There are 
two general categories of school processes: 

* Organizational characteristics pertain to school quality, 
including the norms that the district sets for student 
achievement, teacher behavior, and school operations. 
Some of these organizational requirements are nested 
within state norms, which in turn are nested within 
national norms. Special education policy and procedures 
represent an example of nested organizational character- 
istics. 

* Instructional characteristics include curriculum quality 
(content), teaching quality (pedagogy), time spent in 
instruction (block scheduling, etc.), political processes 
within the school, principal-teacher relations, and stu- 
dent-teacher relations. 
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“Opportunity to learn” (OTL), which is another process 
indicator, means different things to different people. The con- 
cept has evolved since education policy makers first introduced 
it in the 1960s to describe aspects of the education process. 
McDonnell (1995) states that the concept of OTL was developed 
to determine whether cross-national differences in students’ 
mathematics achievement were caused by differences in stu- 
dents’ learning experiences rather than by their ability to master 
the subject. Subsequently, according to Guiton and Oakes 
(1995), as the positive impact of well-designed OTL strategies 
became clearer, they were used to demonstrate how educational 
resources are an equity issue. The construct and measurement 
of OTL have different interpretations; some have argued that 
education outcomes cannot be evaluated without considering 
whether a school has adequate resources and is deploying them 
effectively and equitably. 



Porter (1991) dtes three primary motivations for including 
school process indicators in the education production function: 

• Descriptive motive: Schools provide educational opportu- 
nity, they do not directly produce student learning. 
Therefore, it is important to know about the nature of 
educational opportunity. 

• Evaluation motive: Indicators of school processes can 
serve as evaluation instruments in monitoring school 
reform and a school’s movement toward desired goals. 

• Explanatory motive: Indicators of school processes can 
provide explanatory information when student outcome 
goals are not achieved, pointing to possible causes and 
solutions for inadequacies in school outcomes (p. 14) .C. 

The Challenges in Designing Indicator Systems 

A continual challenge when designing educational indicator 
systems is defining indicators that are: valid, meaning that they 
accurately measure the characteristic of interest; reliable, 
meaning that they will yield the same values under comparable 
data collection measures; and informative, meaning that the 
characteristic being measured is related to a specific outcome 
such as student performance. 
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Figure 1 . A Model of Education Inputs, 
Processes, and Outputs 
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A recent review of educational indicators conducted by the 
National Center for Education Statistics (Mayer, Mullens, & 
Moore, 2000), identified 13 indicators of school quality that are 
considered to be related to student learning. The indicators are 
divided into the categories of school context, teachers, and 
classroom. However, of the 13 indicators, only teacher assign- 
ment, experience and academic skills, and class size were con- 
sidered to be of high quality. These indicators are relatively 
straightforward and have been collected over some time, two 
factors which contribute to the accuracy of indicator data. 

Indicators are powerful political tools and can rouse public 
opinion if misused. In addition, individual indicators are not 
separate entities that can always be categorized as input, 
process, or outcome. An outcome at one stage in the accounta- 
bility process may comprise an input or even a process indica- 
tor at another stage in the process. Finally, appropriate cautions 
must be taken in reporting and interpreting indictors. For exam- 
ple, issues related to comparing indicators across entities versus 
through progress in individuals can have major repercussions. 
Often, a statistically insignificant difference in scores can result 
in a huge difference in ranks. 

Summary 

This overview of educational indicators provides some idea 
of the issues involved in creating such systems. It also provides a 
caution about selecting and using indicators for high stakes 
accountability. It is important to remember that the indicator 
systems do not constitute an entire accountability system in 
themselves. In order for indicator systems to translate into 
accountability measures, a comprehensive system of policies 
and responsibilities must be present. The link between indicator 
systems and accountability systems can be highly tenuous. On 
the one hand, indicator systems can signal trends and direct 
improvement efforts. On the other hand, if indicators are used 
solely for high-stakes decisions associated with sanctions, teach- 
ers may see them as an administrative tool and not a tool for 
school improvement. 
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4. Examples of New Federal Education 

Accountability Systems 



Two pieces of federal legislation that require the use of spe- 
cific indicators as part of enhanced accountability are the 
Government Performance Results Act and Title I of the No 
Child Left Behind Act. They are used here as examples of indi- 
cator systems that call for the collection and reporting of specific 
education data. 

The Government Performance and Results Act of 1993 
(GPRA) 

GPRA (PL 103-62) was passed in 1993- The Act is an 
amendment to the Budget and Accounting Act of 1921, which 
established the Office of Management and Budgeting (OMB) and 
most of the procedures governing financial management of the 
federal government 

The GPRA requires a major culture change in how agencies 
plan their activities and set budget priorities (The Congressional 
Institute, 1996). Prior to GPRA, the focus of evaluation of feder- 
al programs was on activities and regulation - now the focus is 
on outcomes. GPRA holds agencies accountable for producing 
results, and statutorily links plans and results with budgets. If 
agencies do not meet stated goals, their ineffectiveness is docu- 
mented and funding may be cut back or withdrawn. GPRA also 
requires that members of Congress consult with stakeholders to 
discover the relative effectiveness of federal agencies and pro- 
grams. Prior to the enactment of GPRA, agencies would only 
describe the nature and the purposes of their program and 
report the output (i.e., the effort) involved in administering their 
programs. GPRA, however, made an important distinction 
between output and outcomes, and linked inputs to outcomes 
rather than to simple outputs (The Congressional Institute, 1996). 
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Elements of GPRA 

Input - The amount of resources that are devoted to a pro- 
gram activity (e.g. dollars assigned/appropriated for job train- 
ing programs). 

Output - The tabulation, calculation, or recording of activity 
or effort, expressed in a quantitative or qualitative manner 
(i.e., the number of people trained by the job training pro- 
gram). 

Outcome - The assessment of the results of a program activi- 
ty as compared to its intended purpose (i.e., the number of 
people trained by the program who found and kept jobs) . 
GPRA requires that outcome efficacy be measured against a 
set of performance goals through a process of program evalu- 
ation. 

Performance Goal - A target level of performance 
expressed as a tangible measurable objective, against which 
actual achievement will be compared including a goal 
expressed as a quantitative standard, value, or rate. 

Program Evaluation - An assessment made through objec- 
tive management and systematic analysis of the manner and 
extent to which programs achieve intended objectives. 



The Act requires all federal agencies to provide Congress 
with the public sector equivalent of a business plan. Just as cor- 
porations must submit business plans to banks and other finan- 
cial interest-holders, agencies must now produce similar docu- 
ments to be reviewed by agency interest-holders OMB and 
Congress. Specifically, GPRA requires each agency to submit 
three distinct products: 

• A strategic plan covering a period of five years. The first 
plans were submitted to Congress on September 30, 1997, 
and are to be updated every three years. 

• An annual performance plan. The first one was submitted 
to OMB with the FY 1999 budget request and transmitted to 
Congress in February 1998. These agency plans formed the 
basis for a government-wide performance plan, which also 
was submitted to Congress in February 1998. 
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• An annual report on program performance. This is to be 
provided within six months of the end of a fiscal year; the 
first report was due by March 31, 2000. 

GPRA differs from past management reform initiatives in 
two ways. First, it uses the federal budget as a vehicle to provide 
visibility and accountability to the process. By requiring that 
strategic and performance plans be presented in the context of 
an agency’s annual budget submissions, GPRA receives height- 
ened attention and review by executives throughout government. 

Second, in contrast to previous reform efforts, which were 
administrative initiatives, GPRA is the law. Among management 
reform efforts undertaken in this century, only those that have 
been grounded in statute have remained in force (The 
Congressional Institute, 1996). 

Title 1 of the No Child Left Behind Act of 2002 

Although the Elementary and Secondary Education Act of 
1965 (ESEA) has been amended numerous times, the purpose 
of the Title I program, a subsection of the ESEA, has remained 
constant. Title I is designed to close the gap of school achieve- 
ment between wealthy and underprivileged students, to provide 
additional resources to schools serving poor children, and to 
supply equal education and opportunities to all students regard- 
less of socio-economic level (U.S. Department of Education, 
2000). Tide I, Part A is the largest program in the Elementary 
and Secondary Education Act and the recent reauthorization lays 
out an ambitious set of initiatives for schools, some of which 
extend and expand earlier provisions. 

In 1994 the Elementary and Secondary Education Act was 
revamped and several changes were made (U.S. Department of 
Education, 2000). The ESEA was renamed Improving America’s 
Schools Act (IASA). The IASA, along with the Goals 2000 
Educate America Act, built a standards-driven framework for 
reform in schools (U.S. Department of Education, 2001). Under 
this framework Title I required states to create - for all students 
- challenging content and standards aligned with an assessment 
and accountability system, for all students. 



Me I of IASA established several strategic principles to 
ensure that its requirements were followed. States were sup- 
ported in setting high standards for all students and aligning 
curricula, assessments and accountability between Title 1 and 
non-Title I schools. “Upgrading the curriculum, accelerating 
instruction, and providing teachers with professional develop- 
ment to teach to high standards” will focus schools on teaching 
and learning (U.S. Department of Education, 2001, p. 2). 

New Title 1 Federal and State Accountability 
Requirements 

Title I of the No Child Left Behind Act has increased 
requirements for the states, local districts, and schools that 
focus on assessments and accountability for student perform- 
ance. Among the new Title 1 requirements are those that call for 
states to implement annual reading and math testing in grades 
three through eight, and by 2005 in science. States may contin- 
ue to select and design their assessments. However, the assess- 
ments must be aligned with state academic standards and must 
describe three levels of achievement; advanced; proficient; and 
basic. The same standards must be used for all children. 
Furthermore, the assessments must report achievement at the 
child level, be comparable year to year, and be based on meas- 
urable and widely accepted professional assessment standards. 

Assessment results must be disaggregated and reported by 
state, district, and schools by gender, race/ethnicity, English lan- 
guage proficiency, migrant status, low-income status, and special 
education status. The goal is to bring every child to proficiency 
within 12 years. To accomplish this, the Act sets a precise 
timetable for schools and district improvement (H.R.I., The No 
Child Left Behind Act, Conference Agreements, httpredwork- 
force.house.gov/issues/107th/education/nclb/). 
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Htle 1 of NCLB 

• Lowers the poverty threshold for schoolwide programs to 
40 percent; 

• Requires standards and assessments in science; 

• Requires states to annually test the proficiency of limited- 
English proficient students; 

• Institutes diagnostic testing for literacy in first grade; 

• Specifies that "adequate yearly progress” address the 
progress of all disadvantaged subgroups; 

• Requires all students to be taught by "highly qualified 
teachers"; 

• Requires Ming schools to offer both school choice and 
supplemental services, such as tutoring; 

• Requires funds to be used to promote "scientific, research- 
based and effective practice in the classroom"; 

• Requires districts to report teacher qualifications to parent 
on request; and 

• Adds substantial other new reporting requirements. 

(Source: Educating Funding Research Council, January, 2002, p. 9). 



A key accountability tool is Adequate Yearly Progress (AYP) 
that states must define for all local districts and schools. The 
AYP measure must include the performance of all students and 
requires that states establish their baseline “based on the higher 
of the percentage of the students at the proficient level who are 
in either the lowest achieving group of students, or the school at 
the 20th percentile in the state”(Educational Funding Research 
Council, January, 2002b, p. 7). 

The AYP must include separate, measurable annual objec- 
tives for all public school students and for each of the key 
groups identified above. The AYP must include graduation rates 
for secondary school students and at least one other academic 
indicator. The AYP must include intermediate performance 
goals in addition to the proficiency “target” set by the state in 
order to gauge the progress of various groups. The AYP for 
individual schools, districts, or the state may only be achieved if 
95 percent of each subgroup is tested. The Act also specifies 
consequences for schools that M to meet prescribed levels of 
performance (Educational Funding Research Council, January, 
2002b). 

Title 1 schools that M to achieve AYP for two consecutive 
years are placed in corrective action and identified for school 
improvement and must offer students an option to transfer to 
another public school, notify parents, develop prescribed school 
improvement plans, offer specific “supplemental services” from 
a “provider of choice”. The Act defines additional explicit 
actions that local districts and states must take with individual 
schools placed in corrective action (Educational Funding 
Research Council, January, 2002b, p. 7). 

The assessment and accountability requirements of the No 
Child Left Behind Act are accompanied by a number of other 
new provisions that address teacher quality and funding. The 
provisions take effect immediately, unless otherwise specified, 
and represent a more forceful and comprehensive federal role 
in school accountability. 
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A. Hie Current Status of Title I State Accountability 

Systems 

The U.S. Department of Education determines whether a 
state has met the Title I requirements. The Department conduct- 
ed peer reviews, involving experts in the fields of standards and 
assessments to ascertain if states were meeting the 1994 IASA 
Title I assessment requirements. The peer review did not direct- 
ly examine a state’s assessment, but reviewed materials submit- 
ted by each state to provide evidence that it met the require- 
ments. The areas of review included general characteristics of 
the assessment, alignment with content standards, technical 
quality, and reporting and use of results for accountability (U.S. 
Department of Education, No Date) . Each state’s assessment sys- 
tem was judged to be in one of the following categories: full 
approval, conditional approval, compliance agreement, timeline 
waiver, or under review. 

B. State Variations in Interpretation. 

Title I has the same requirements for all states regarding 
assessments, standards, performance reporting, and conse- 
quences for performance. The general nature of the 1994 
requirements resulted in a variety of state-level interpretations, 
reflecting state demographics, political culture, educational per- 
formance, and educational governance structures and policies 
(Goertz & Duffy, 2000). 

A primary intent of Title I of IASA was that each state would 
develop a rigorous accountability system that would apply equal- 
ly to Title I and non-Title I students and schools (U.S. 
Department of Education, 2001). A review of state assessment 
and accountability systems conducted by the Consortium for 
Policy Research in Education (Goertz & Duffy, 2001) concluded 
that many states are still functioning under dual accountability 
systems with differing standards for Title I and non-Tltle I 
schools and different measures of adequate yearly progress. 

This means that there is considerable annual fluctuation in the 
percentage of schools identified as “in need of improvement” 
within any given state (U.S. Department of Education, 2001). 



Another major area of state variation related to assessment. 
Assessment was identified as the foundation of the states' 
accountability systems. States were to use multiple measures to 
assess their students. However, what is meant by multiple meas- 
ures was not precisely defined, which led states to translate this 
requirement to fit their own needs or existing systems (Goertz & 
Duffy, 2000). States are using norm-referenced, criterion-refer- 
enced, and performance-based tests solely or in combination 
and some states are using assessments that are not aligned with 
their challenging content standards or performance standards 
(U.S. Department of Education, 2001). 

Title I of IASA stated that all students were to be held to the 
same assessment, reporting, and accountability policies. As of 
January 2001, a total of 20 states had not met the Title I require- 
ments to include students with disabilities or students with IEPs 
in their assessment and accountability systems (U.S. Department 
of Education, 2001); states also differed in the way students with 
limited English proficiency were included. Often, state assess- 
ments did not provide valid and comparable measures for stu- 
dents with disabilities or with limited English proficiency 
(Goertz & Duffy, 2000) . 

A final concern regarding state interpretation of the 1994 
requirements was the lack of states' capacity to support the 
improvement of low-performing schools. Many states have had 
to limit the number of schools identified as low performing 
because funds do not allow for intervention at all the schools in 
need (Goertz & Duffy, 2000). For example, “California designat- 
ed 3,144 schools as under-performing in 1999-2000, but 
included only 860 of these schools in the first two years of its 
Immediate Intervention/Under-performing Schools Program” 
(Goertz & Duffy, 2000, p. 39). As noted earlier, the new reau- 
thorization of Title I strengthens both the accountability require- 
ments and support for school improvement. Nonetheless, given 
the status of the implementation of the previous requirements, a 
number of states may have a way to go to bring their assessment 
and accountability system fully into compliance with the new 
requirements. 
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5. Special Education Accountability 

and Indicators 



Accountability for special education programs has tradi- 
tionally differed from general education accountability in at least 
two major ways. First, the focus in special education has been 
overwhelmingly on compliance with legal procedures (Elmore 
& McLaughlin, 1982; Thurlow et al., 2000; U.S. Department of 
Education, 1998). Second, accountability for student perform- 
ance or outcomes has been private, through the Individualized 
Education Program (HP) review process, as opposed to public 
through reported aggregate data (McDonnell, McLaughlin, & 
Morison, 1997; McLaughlin, Henderson & Rhim, 1997). The 
strong federal focus on accountability through compliance mon- 
itoring reflects the civil rights orientation of ensuring equitable 
access to a free and appropriate public education (EAPE) that 
drove the creation of a special education system in the U.S. 
(Danielson & Malouf, 1994; McDonnell et al., 1997). 

The National Council on Disability (NCD, 1989) suggested 
that the focus in special education should shift from access to 
education to the quality of education and student outcomes. 
Several years later, a National Research Council (NRC) study 
examined how students with disabilities were to be included in 
standards-based reforms (McDonnell et al., 1997). The NRC 
endorsed two principles regarding students with disabilities and 
standards-based reform: 1) All students should have access to 
challenging standards; 2) policymakers and educators should 
be held publicly accountable for every student’s performance. 

Special Education Accountability Frameworks 

The National Center on Educational Outcomes (NCEO) and 
the National Association of State Directors of Special Education 
(NASDSE) have offered models of accountability for special edu- 
cation that include specific indicators. In the early 1990s, NCEO 
worked with hundreds of stakeholders to develop a conceptual 
model of educational results and indicators to guide accounta- 
bility for students with disabilities (Ysseldyke, Krentz, Elliott, 
Thurlow, Erickson, & Moore, 1998). This framework is also 
used to design alternate assessment systems and guide IEP plan- 
ning for students with disabilities, ensuring that individual goals 
are comprehensive and aligned with state or district standards. 

The NASDSE Balanced System of Accountability (NASDSE, 
1995) builds on a conceptual framework that assumes a dynam- 
ic balance between the following three components: inputs and 
processes that guarantee educational equity (e.g., IEP, least 
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restrictive environment, procedural safeguards, parent involve- 
ment, staffing credentials, finance/funding, and staff develop- 
ment); system results that guarantee program effectiveness (e.g., 
standards, curriculum, sanctions, and incentives); and individ- 
ual student learning (e.g., IEP goal attainment, academic and 
non-academic performance, and continuous monitoring). The 
value of these NCEO and NASDSE models largely rests with the 
articulation of key student outcomes and performance indica- 
tors that can guide state and local education improvement plans. 

New IDEA Accountability Requirements 

The 1997 IDEA Amendments contain a number of new pro- 
visions that seek to align federal special education policy with 
general educational accountability reforms. The SEAs must 
ensure that children with disabilities are included in local and 
statewide assessments with accommodations where appropriate 
[Section 6l2(a)(17)(B)(I)], and report the performance of 
these children with the same frequency and in the same detail 
that they use to report non-disabled children performance levels 
[Section 6l2(a)(17)(B)(iii)]. 

States are also required to develop alternate means of 
assessment for those children who are unable to participate in 
standard assessments by the year 2000 [Section 
6l2(a)(17)(A)], and the performance of these children must 
be reported as well [Section 6l2(a)(17)(B)(iii)]. In addition, 
IDEA requires states to create performance goals and indicators 
for children with disabilities that are aligned with the states’ 
established learning standards and desired educational out- 
comes for all children. According to IDEA, the goals for chil- 
dren with disabilities: (1) are required to be “consistent, to the 
maximum extent appropriate, with other goals and standards for 
children established by the state;” (2) must “at a minimum 
address the performance of children with disabilities on assess- 
ments, dropout rates, and graduation rates;” 3) and shall be 
part of the report “to the Secretary and the public on the 
progress of the State, and of children with disabilities in the 
State, toward meeting the goals...." [(Sec. 612 (a)(l6)]. 

By adding these provisions, the law implicitly defines 
state and local assessments as contributing to a student’s educa- 
tional opportunities, for which access must be guaranteed. Tying 
progress to local and statewide assessments, with appropriate 
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accommodations, can ensure that students with disabilities will 
access the same standards as all other students. The require- 
ment for public reporting of aggregated data is designed to 
enforce accountability and the educational progress of students 
with disabilities. 

While the IDEA contains the most explicit statements con- 
cerning educational accountability, provisions in both Section 
504 of the Rehabilitation Act (1973) and the Americans with 
Disabilities Act (1990) also provide important guarantees relat- 
ed to educational reform. 

According to a recent analysis of state education reports 
conducted by NCEO, 33 states publicly reported data on the par- 
ticipation of students with disabilities in assessments and 37 
publicly reported data on the assessment performance of stu- 
dents with disabilities (Thompson & Thurlow, 2001). While 
these numbers have increased considerably from previous 
years, still only 13 states reported participation data on students 
with disabilities for all of the tests for which they reported par- 
ticipation data on other students, and only 19 states reported 
performance data on students with disabilities for all of the tests 
for which they reported performance data on other students. 

See “Reporting on the Performance of Students with Disabilities, 
www.eppri.org. 

While students with disabilities are increasingly being 
accounted for in general education reports, several efforts with- 
in special education have been specifically designed to provide 
OSEP with information related to how state education agencies, 
state lead agencies for early intervention, local education agen- 
cies, early intervention service providers, and other relevant 
state agencies are complying with the IDEA federal mandates. 

These include the annual state reported data, OSEP monitoring 
activities, and biennial reports. 

Annual Report to Congress on the Implementation of 
IDEA 

Since the implementation of PI. 94-142 (1975), states have 
been required to annually report data on a variety of program 
indicators including the number of students receiving special 
education services, their placements, the number of personnel 
serving them, and the exit status of those leaving school. Sec. 

618 of the IDEA specifies the data reporting requirements of 
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states. The data must include the following items, although the 
Secretary of Education may request other data related to the 
implementation of the IDEA. (The items here pertain to Part B 
or school age students. Data are also required for preschoolers, 
infants and toddlers.) 

• Child Count-the number of students with disabilities receiv- 
ing special education and related services under IDEA. 

• Educational Environments- the extent to which students with 
disabilities are being served with their non-disabled peers. 

• Personnel-the number of personnel employed and contract- 
ed to provide special education and related services. 

• Exiting-the number of students, ages 14-21, who exit spe- 
cial education during school. 

• Disdpline-the number of students who are placed in an 
interim alternative education setting, the acts precipitating 
those removals, and the number of children who are sub- 
ject to long-term suspensions or expulsions. 

• Other data as required by the secretary- specific data, child 
count and settings, disaggregated by race and ethnicity. 

The annual reports, which have been published since 1978, 
contain the state-reported data and have been some of the main 
sources for reporting the nation’s progress on implementing the 
IDEA (U.S. Department of Education, 1998) . Some of the data, 
such as numbers of students served under the IDEA, have been 
collected since the beginning; other data, such as on suspen- 
sions and expulsions, are more recent requirements. In addition 
to the data, the reports include information from OSEP monitor- 
ing activities, and discretionary programs such as research. 

Currently, the Annual Reports to Congress are divided into 
four main sections: Context/Environmental, Student 
Characteristics, School Programs and Services, and Results. 

Each section contains individual modules on a variety of topics 
of interest to the special education field, and provides an 
overview of current issues affecting students with disabilities. 
Appendices in the reports contain the state data tables. 
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Significant efforts have been made to create greater unifor- 
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mity in the data and permit cross-state comparisons. However, 
much variability exists across states because of differences in 
state definitions of certain data categories. Generally, the longer 
a specific data element has been reported, the more stable and 
consistent die indicator. 

OSEP’s Monitoring Plan 

As the adminis trative agency responsible for overseeing the 
implementation of IDEA, OSEP assists state agencies through 
grants, technical support, policy support, and monitoring over- 
sight. Each SEA is charged with ensuring that local schools 
implement the federal provisions (as well as individual state 
requirements). Traditionally, the monitoring process employed 
by OSEP has focused on determining compliance with various 
procedures required under the IDEA. These include having a 
system in place to identify children with disabilities, instituting 
non-discriminatory assessments, meeting the prescribed time 
lines for specific procedures, and other processes and activities 
mandated by federal special education law and regulation. 

A. Continuous Improvement Monitoring Process 
(CIMP) 

Since the 1997 reauthorization of IDEA, OSEP has been 
moving toward outcome- or performance-based monitoring 
focused more on results than on process. Program administra- 
tors from OSEP engaged in discussions with stakeholder groups 
of parents, teachers, advocates, and representatives of state and 
local school districts to design a different monitoring process. 
The collaboration between OSEP and these key stakeholders led 
to the development of the Continuous Improvement Monitoring 
Process (CIMP) (OSEP, 2000). 

The CIMP has been designed to assess “the impact and 
effectiveness of State and local efforts to provide early interven- 
tion services to infants and toddlers with disabilities and their 
families, and a free appropriate public education to children 
and youth with disabilities” (OSEP, 2000, p. 3). It employs a 
multi-step process eng ag ing a stakeholder committee at the state 
level to assess critical program needs through the identification 
of key program indicators. 

The first phase consists of a state-level “self-assessment” 
which is completed by a steering committee of key stakeholders. 







The committee constructs and employs a self-assessment plan to 
evaluate how well the state has achieved compliance and 
improved results for students with disabilities (OSEP, 2000). 
Self-assessments are organized around five cluster areas for Part 
B (school-age programs) and Part C (infant and toddlers pro- 
grams) . Part B clusters areas are: parent involvement; free and 
appropriate public education in the least restrictive environ- 
ment; secondary transition; early childhood transition; and gen- 
eral supervision. The Part C areas are: family centered services; 
child find; early intervention services; and general supervision. 
For each cluster area, there is an overall objective, a compo- 
nent, indicators, and recommended data (Ryder, 2001). 

During “validation planning,” which follows the assessment 
phase, the committee and OSEP collaboratively corroborate the 
self-assessment Strategies include reviewing collected data 
and/or collecting additional data, and conducting on- and off- 
site reviews, public meetings, and interviews. 

“Improvement planning” takes place after the steering 
committee has reviewed its validation results and self-assess- 
ment. The committee develops an improvement plan that focus- 
es on compliance with the law and the improvement results for 
students with disabilities. Improvement methods in the plan 
must include timelines and benchmarks. Following development 
of the improvement plan, the strategies are implemented and the 
results evaluated. 

The final phase in the CIMP is “verification and conse- 
quences,” which is OSEP’s verification of the effectiveness of the 
state’s plan based on data and documentation supplied by the 
state and its steering committee. The state may be rewarded for 
its successful approaches to complying with IDEA, although it is 
not clear what rewards are given. States that do not comply, 
have not implemented mandatory components, or have unsuc- 
cessful improvement plans may have to take corrective actions, 
enter a compliance agreement, or have funds withheld (OSEP, 
2000 ). 

B. Focused Monitoring 

In November 2000, the cluster areas were reviewed by the 
stakeholder groups originally involved in developing the CIMP, 
and further revisions were made. Specifically, the deliberations 
resulted in a move to “focused monitoring.” The key principles 
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of focused monitoring indude defining the following: priority 
areas; indicators measuring performance in each priority area; 
benchmarks that compare state data to the state’s own baseline 
performance; and triggers or performance levels that determine 
whether a state needs further assistance or intervention or has 
developed a successful practice that can be shared with other 
states (Ryder, 2001). 

Focused monitoring is based on the assumption that there 
are limited resources available to OSEP and to state education 
agendes to assist school districts in the implementation of IDEA. 
Therefore, resources should be allocated where there is the 
greatest need; resources should not be allocated unless they are 
suffident to guarantee the desired outcome. Monitoring should 
focus on select priorities that are likely to improve student edu- 
cational performance and increase the independence of chil- 
dren with disabilities. Focused monitoring is also based on the 
belief that to be effective, monitoring should be limited to a 
number of priority areas and a number of indicators within 
each area. The indicators should be verifiable and able to target 
OSEP’s and state resources in the direction of greatest need. At 
this time, focused monitoring is only a proposal, and key indica- 
tors are being discussed and debated. 

Requirements Regarding Performance Goals and 
Indicators 

The 1997 IDEA amendments contained a new requirement 
that states establish performance goals and indicators for stu- 
dents with disabilities and report to the public on progress 
made toward meeting these goals. States report in a “Biennial 
Performance Report for Part B (of IDEA)." The first reports 
were submitted on December 31, 1999- 



Each state was required to provide the following informa- 
tion about its performance goals and indicators: “State the goals 
the State has established for the performance of children with 
disabilities in the State and the extent to which those goals are 
consistent with other goals and standards for non-disabled chil- 
dren established by the State.” Also, “State the performance 
indicators that the State will use to assess progress towards 
achieving those goals that, at a minim um, address the perform- 
ance of children with disabilities on assessments, drop-out 
rates, and graduation rates” (Aheam, 2001, p. 1). 

A review of the first Biennial Performance Reports revealed 
significant variation across states. For instance, states’ special 
education performance goals and indicators are at very different 
stages of development Ten of the reports were found to be 
incomplete while others were very detailed. Some states had 
specific indicators listed with broad goals and vice versa. What 
one state may have distinguished as an indicator, another state 
may have identified as a goal (Ahearn, 2001). 

About one-third of the states applied the same set of goals 
to students in special education as in general education, while 
the remaining states had separate sets of goals for each group. 
The most common goals, among all states, addressed improving 
academic achievement Other goals, in order of frequency, 
addressed transition or postsecondary placements, teacher 
preparation and technical assistance, graduation rates, dropout 
rates, and communication or coordination with families and 
community (Ahearn, 2001). 

Variations in state performance indicators mirrored varia- 
tions in state goals. What were goals in one state were indicators 
in the next. The number of indicators reported by states ranged 
from 5 to 71, and were clustered in the following eight major 
areas: graduation standards; inclusion in general education cur- 
riculum and/or assessments; improvement of dropout rates; 
raising academic achievement; improving transition and post- 
school plans; decrease of suspension and expulsion rates; high- 
er quality of service for teachers and other personnel; and 
greater communication and coordination with families (Ahearn, 
2001 ). 
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Summary 

The previous sections have overviewed some of the consid- 
erations in creating indicator systems, particularly those that are 
to be used as part of accountability systems. Additionally, exam- 
ples were presented of current federal-level approaches to pro- 
gram accountability that use a variety of performance indicators. 
Several key conclusions should be drawn from the overview. 
First, it is important to restate that creating an indicator system 
does not directly lead to accountability. Indicators can have mul- 
tiple uses, one of which is accountability. Indicators can and 
should be able to be used to monitor program improvement 
and identify “hot spots” of underperformance. However, the 
purpose and use of a specific indicator or group of indicators 
should be defined first, then indicators selected. Simply using 
data that are available or convenient may not serve the purpose 
of the indicator system. 

A second consideration concerns building an indicator 
“system” that has a linkage between individual indicators. 

Ideally only data that has a direct linkage to a specific outcome, 
such as student performance, should be collected as part of an 
indicator system. However, given the difficulties in measuring 
educational processes and the weak relationships between some 
of the more reliably measured indicators and student progress, 
this may be a challenge. Nonetheless, simply collecting and 
reporting an assortment of statistics can lead to random acts of 
improvement 



Related to the notion of building a system of indicators is 
the idea of having vertical alignment within the system. Federal 
level data, in most instances, is reported up Grom school, dis- 
trict, and/or state. Thus, consideration should be given to build- 
ing a system of individual statistics or data elements that have 
use and meaning at the level at which they are reported as well 
as up through the system. That is not to say that the only indica- 
tors should be those reported to the state or federal levels. It 
does argue for alignment so that federal and state-level deci- 
sions or actions have some probability of impacting at the spe- 
cific point in the system that needs to be remedied. 

Last, but certainly not least, an indicator system should be 
“parsimonious” and die data elements used as indicators should 
be chosen with care. In addition, efforts should be made to 
ensure that the data is of the best quality possible. In die follow- 
ing section we discuss in detail how to think about creating an 
indicator system and choosing specific indicators. 
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Table 1 . Status of Core States' Performance Goals 

and Indicators (as of 1 1/2000) 



State is California 




State is Maryland 




Goals 


Indicators 


Goals 


Indicators 


1. Unique needs of students 


1. Percent of students partid- 


1. Expand opportunities for 


1. Average scores for students 


are to be accurately identi- 


paling in the 


students with disabilities to 


with disabilities will improve 


fled. 


assessment program. 


participate in the general 
education curriculum and 


by 1.5% annually. 


2. Have fully qualified per- 


2. Percent of students over the 


school reform. 


2. The percent of time stu- 


sonnet. 


fiftieth percentile in assess- 




dents with disabilities spend 




meals. 


2. Increase the inclusion of 


in regular education will 


3. Integration. 


3. Average reading scores at 


students with disabilities in 
measures of school and 


increase by 2% per year 


4. Have high academic stan- 
dards. 


fourth grade. 


student performance. 


3. Percent of students with 
disabilities vtho receive a 


5. Prepare students with dis- 


4. Percent of students who 


3. Enhance partnerships. 


high school diploma will 


drop-out during or after 




increase by 2% annually. 


abilities for work and 


eighth grade. 


4. Strengthen the professori- 


independence. 


5. Percent of special educa- 
tion students returned to 


al work force require- 


4. Increase post high school 




ments. 


employment by 2% per year. 




general education. 

6. Percent of twelfth grade stu- 
dents who graduate. 


5. Coordination improve- 
ment 

6. Support system change 
activities, coordinate 


5. No students will be exempt- 
ed from assessments by 
2001. 

6. Percent of students with 




7. Four measures of 


research, personnel 


disabilities who are 




ethnic disproportion. 


preparation, technical 
assistance, and media 


expelled, suspended, 
and/or highly challenging 




8 Percent of qualified 
personnel 

9- Percent of students 
expelled. 


changes. 


behavior will decrease by 
10% per year. 

7. Decrease die placement of 
students with disabilities in 
alternate environments 
because of challenging 
behavior by 10% per year. 

8. Increase students’ with dis- 
abilities attendance rates by 
2% per year. 

9- Decrease dropout rate of 
students with disabilities by 
0.5% per year. 
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Table 1 . Status of Core States 7 Performance Goals 

and Indicators (as of 1 1 /2000) 







State is New York 


State is Texas 



Goals 



Indicators 



Goals 



Indicators 



(Same goals for AIL 

students) 

1. All are to meet high aca- 
demic standards and 
behaviors. 

2. Regents standards are to 
be met by all schools. 

3. All personnel must be 
qualified and current. 

4 All education and 
resources must be accessi- 
ble to everyone. 

6. Work environment must 
meet high standards. 



1. Increase the percent of stu- 
dents with disabilities in 
state assessments to 95%. 

2. Decrease the percentage of 
dropouts. 

3. Have 80% of students with 
disabilities who complete 
their secondary education 
receive a diploma. 

4. No more than 20% of stu- 
dents with disabilities 
receive an IEP diploma or 
local certification. 

5. Resources are to be used in 
public interest 



(Same as State goals for 

ALL students) 

1. Exemplary performance in 
reading and writing. 

2. Exemplary performance in 
understanding of mathe- 
matics. 

3. Exemplary performance in 
understanding of science. 

4 Exemp lary performance in 
understanding of social 
studies. 



(Same as State Indicators) 

1. Results on assessments. 

2. Dropout rates. 

3- Attendance rates. 

4. Percent of students who 
pass tests per grade. 



Note. From Abeam, E. (2001). Performance Goals and Indicators. 
Washington D.C.: National Association of State Directors of 
Special Education. Adapted with permission of the author. 
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6. Issues in Creating Special 
Education Indicators 



The overarching purpose of an education indicator sys- 
tem is to permit refined and balanced judgments about the 
quality of education and, to the extent possible, a better 
understanding of the relationships between actions by policy 
makers and changes in that quality. ...The mere publication, 
however, of a single indicator or collection of indicators is 
generally insufficient to describe or explain the condition of 
something as complex as an educational system (Goertz, 
1989, p. 3). 

There has been a tendency to use budding state indica- 
tor systems to hold local school systems, schools, and school 
staff accountable for results before the system has the capa- 
bility for adequately doing so ( Goertz , King, Coley, & Kaagan, 
1989, cited in Goertz, 1989, p-3)- 

In this section we lay out some of the basic steps in devel- 
oping special education indicator systems and performance 
goals. 

Steps to Creating A Special Education Indicator System 

There is no absence of special education data, at least at 
the state and federal levels. The difficulties facing special educa- 
tion policy makers lie in making sense of the existing data and 
determining which data can be used as indicators. A special 
education indicator system is an organized set of statistics the 
can inform policymakers, administrators, parents, and other 
stakeholders about key aspects of the special education system. 
Defining an indicator is more than simply choosing a statistic. 
While we may cherish certain statistics or “data”, we must also 
understand what the statistic “indicates.” In considering how to 
select indicators, there are several important considerations. 

First, we must think carefully about the overall purpose of 
the indicator system. Then we must consider how the properties 
of the data match the decisions to be made. Of course, all of 
these decisions are balanced with issues of cost and efficiency. 



U 




A. Purpose of the Indicator Systems 

One use of indicators is simply to describe some compo- 
nent of special education (e.g., performance of students with 
disabilities, quality of personnel, extent of use of certain set- 
tings). The data can also be used to analyze the relationships 
among components (e.g., does student performance differ by 
setting or teacher qualification?). Ideally, a special education 
indicator system should be both descriptive and analytical. 
However, one that is purely descriptive is only useful if it pro- 
motes further investigation or systematic analyses. Further, to be 
useful, an indicator should be trackable over time and reveal 
trends. 

The definition of an indicator system’s purpose is entangled 
with the kinds of decisions that will be made based on the data. 
The decisions, in turn, are related to the level at which actions 
will be taken. 

Table 2 contains examples of how data are used at federal, 
state and local levels and considerations for choosing indica- 
tors. 
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Table 2. Considerations for Choosing 
Indicators Based on Use 
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B. Choosing Indicators 

Indicators may likely differ depending on the stakes or con- 
sequences of decisions (e.g., resource allocation, capacity 
building, corrective action) and the level (student, school, dis- 
trict, state, or federal) at which they are made (see Figure 2). 
Qeariy, indicators reported to the federal level may not need the 
same properties as an indicator that might be used by a special 
education director to identify “hot spots” of poor performance 
in schools. Also, if consequences are attached to the data, such 
as further investigation or sanctions, the stakes increase and the 
properties of the indicator become even more critical. 

Indicators may focus on key outcomes and processes. 
However, data on context and inputs are also necessary to per- 
mit investigation. 

Key questions that must be considered in selecting indica- 
tors are: is the indicator representative of the specific element of 
interest and not confounded by other elements; how stable is 
the indicator over time; can the indicators be aggregated up and 
compared across schools, school districts, or states and 
“benchmarked” to other similar data (e.g., general education, 
other states, or agencies). 

C. Understanding “Validity” of Indicators 

During the last two decades, as our understanding of edu- 
cational measurement technology has evolved, so too have our 
notions of validity. Our understanding has moved beyond the 
narrow conception that validity rests in a particular measure- 
ment tool. It is now generally understood that validity requires 
an evidential basis for making judgements or decisions 
(Messick, 1989). Tests are not valid or invalid per se; rather it 
is the inferences that are made on the basis of data obtained 
from a test that are valid or invalid. 



all similar or related inferences. For example, data that show a 
high rate of participation of students with disabilities in a 
statewide assessment program would not necessarily support 
the inference that students on IEPs are making progress in the 
general education curriculum. On the other hand, some forms 
of data can be used effectively in multiple contexts to support 
various decisions. Scores obtained from nalionally-normed 
achievement tests may be used to support decisions in the class- 
room (e.g., develop reading groups), the district (select reading 
curriculum materials), and on the state level (allocate funds to 
districts with low reading scores) . 

Establishing validity involves two types of evidence: data that 
lend credence to the inference being made, and an absence of 
data to support competing or alternative hypotheses. 

Information from multiple sources must converge on the con- 
clusion that the decision or course of action is valid. Validity is a 
relative, rather than absolute entity. 

Ideally, the higher the stakes associated with an indicator, 
the stronger the body of evidence needed to validate decisions 
made on the basis of the indicator. For example, decisions 
based solely on test scores would be suspect in the absence of 
key information about participation rates, opportunities to 
learn, or resources, as well as other outcome or performance 
data. 



This conception of validity is particularly relevant when 
considering the properties of indicators. By definition, indica- 
tors are used to make judgments and decisions. However, deci- 
sions may depend more on the context in which they are made 
than on the properties of the indicators. 

Validity involves the interpretation of information in a par- 
ticular context. An indicator that supports the validity of one 
decision would not represent an adequate evidentiary basis for 
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Figure 2: Choosing Federal State, 
and local Indicators 
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D. Guidelines for Selecting Indicators That Support 
Valid Decisions 

The following are considerations for selecting indicators 
that support valid decision-making: 

• All information is not of equal value. Indicators should be 
selected on the basis of their relevance to the specific deci- 
sions being made. Collecting better information is more 
desirable than simply collecting more information. 

• To the extent possible, indicators should be selected that 
directly relate to the decisions being made. The lower the 
level of inference required to interpret data associated with 
an indicator, the more likely it will support valid decisions. 

• Decisions are likely to be more valid when they are based 
on multiple sources and types of information that converge. 

• Indicators that lack stability (over time, across groups, 
across settings, etc.) are of limited use for valid decision- 
making. 

• Information has different meanings and value at different 
levels. For example, classroom teachers may attach differ- 
ent meaning to a particular piece of information than would 
a district administrator or representative of an SEA. 
Therefore, indicators must be selected on the basis of the 
level of decision-making they will support. Particular care 
must be exercised when data collected in the context of one 
level are aggregated or disaggregated for decision-making 
at another level. 

• information loses value over time. To the extent possible, 
indicators should be selected that provide information in 
“real time” to the decision-makers who will be using it so 
that improvements can be made in a timely manner. 

E. Efficiency of Data Collection 

The concept of efficiency refers to increasing an impact or 
outcome without increasing the level of required resources, or 
maintaining a level of outcome while using a lower level of 
resources (Hill, Pierce, & Guthrie, 1997). With respect to 
selecting indicators, efficiency is an important concept The 
costs of collecting and analyzing data associated with an indica- 
tor must be weighed against the extent to which the indicator 







leads to more valid decision-making. Two potential problems 
can arise. As a hedge against making poor or unpopular deci- 
sions, policymakers may err on the side of collecting more 
information than is necessary. Or if a statistic is not particularly 
informative and requires extensive “digging” and further data 
collection, efficiency is lost 

As data elements are added to the decision process, more 
resources must be allocated and teachers and parents may 
believe there is “too much testing,” or district or state-level 
administrators may perceive “too much monitoring.” As a 
result, policy- makers may try to make decisions on the basis of 
too little or bad information. Collection of more pertinent infor- 
mation may appear to be too expensive or difficult, so the poli- 
cy-makers use information that is readily available. 

Unfortunately, the data may actually provide minimal support for 
the decisions, particularly for high stakes decisions. 

Collecting data that has limited use is not only inefficient; 
there is limited motivation to ensure the quality of the data. On 
the other hand, minimizing cost factors for data that have limit- 
ed value is also inefficient. 

Certain factors can create more efficiency in the indicator 
system: 

• Valid and reliable data are needed for valid decision-mak- 
ing. They may cost more to collect, particularly initially, but 
will likely have more impact because they lead to meaning- 
ful changes. 

• Removing redundancies in data collection creates a more 
efficient system. Using data collected by the system and/or 
for another program, such as Title 1, and then disaggregat- 
ing students with disabilities is more efficient than collect- 
ing new data or creating new variables. 

• Data collection should be coordinated both horizontally, 
across various programs and offices, and vertically, from 
school to district to state. This may contribute to redundant 
data collection, but is still increasingly important as differ- 
ent educational programs move toward performance-based 
accountability and data-driven decision making. As more 
data are required, there is greater need for a coordinated 
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indicator system and data collection efforts. To the extent 
that federal indicators are useful at state and local levels, 
there will be greater efficiency and accuracy in the system. 

• Standardizing the operational definitions of specific indica- 
tors is important for comparability and coordination and 
can lead to data collection forms and reports that are useful 
at each level of the special education system. 

• Finally, all those who will use the indicators for decision 
making should value them. A statistic must have meaning 
and use, or those responsible for collecting and reporting 
will have little incentive to collect and record accurately. 

Since the vast majority of special education data must be 
collected at the school and/or local district level, the most criti- 
cal indicators need to be meaningful at those levels. To achieve 
this, the indicator system should not be developed in isolation of 
local needs, nor represent data “silos” with each program col- 
lecting and reporting its own isolated data sets. Only when the 
system has alignment and coordination will data collection be 
efficient and meaningful. 

Conclusions: Putting it All Together 

Blank (1993) and Goertz (1989) have suggested ways of 
designing educational indicator systems. Their process is 
reflected in the following key decisions and actions useful for 
special education policy-makers and administrators: 

• Develop a conceptual framework based on a clear purpose. 
Is the intent of die indicator system to measure some edu- 
cational phenomenon or to compare educational phenome- 
na in different school districts or states? 

• Identify the phenomenon of interest. What are the important 
features of the educational system that might be captured? 



• Align indicators with key goals of the program and align the 
indicator system with other data collection efforts 

if possible. 

• Obtain commitment and co-operation of leaders: does the 
political climate of the state or district create a conducive 
environment for the indicator system? 

• Involve policy makers, educators, researchers, and data 
managers in selecting data. 

• Be parsimonious. Select a limited number of indicators and 
minimize complexity in reporting. 

• Be clear about measurement levels. At what levels should 
data be collected (federal, state, local)? How can compara- 
bility be assured with data collected by different agencies 
with different collection instruments and varying data defi- 
nitions? How can quality control be main tained? 

• Decide up front how and to whom reports will be made. 

This review has covered the theoretical and practical 
aspects of creating indicators in special education. We also rec- 
ognize that there are a number of elements that need to come 
together to achieve the indicators we propose. However, the 
benefits of such an efficient and aligned system of data collec- 
tion merit serious consideration. Further, the current climate of 
enhanced performance-based accountability should provide the 
impetus for its development. 
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